Watermarking to control video recording

ABSTRACT

Video programs are identified by watermark data subliminally conveyed within the picture data. By reference to such program identification data, video recording devices can automatically identify desired programs, and respond accordingly. For example, if a desired television program is originally scheduled to air at 5:00 p.m., but is delayed until 5:10 due to a sporting event that exceeded its broadcast slot, a video recorder equipped with the disclosed technology can sense the delayed start, commence recording at 5:10, and continue recording until the (delayed) end of the program. The program identification can also identify the program&#39;s subject matter, allowing automated recording of programs that match a viewer&#39;s profile.

RELATED APPLICATION DATA

[0001] This non-provisional application claims priority benefit toprovisional application No. 60/303,173, filed Jul. 5, 2001.

[0002] Filed on the same day as this application are two companionapplications, both claiming priority benefit to application No.60/303,173. One is entitled Watermarking and Electronic Program Guides.The other is entitled Watermarking to Set Video Usage Permissions.

FIELD OF THE INVENTION

[0003] The present invention relates to digital watermarking, and moreparticularly relates to use of watermarks in connection with startingand stopping of video recording.

BACKGROUND

[0004] Digital watermarking is the science of encoding physical andelectronic objects with plural-bit digital data, in such a manner thatthe data is essentially hidden from human perception, yet can berecovered by computer analysis. In physical objects, the data may beencoded in the form of surface texturing, or printing. Such marking canbe detected from optical scan data, e.g., from a scanner or web cam. Inelectronic objects (e.g., digital audio or imagery—including video), thedata may be encoded as slight variations in sample values. Or, if theobject is represented in a so-called orthogonal domain (also termed“non-perceptual,” e.g., MPEG, DCT, wavelet, etc.), the data may beencoded as slight variations in quantization values or levels. Thepresent assignee's U.S. Pat. Nos. 6,122,403 and 6,408,082, andapplication Ser. No. 09/503,881, are illustrative of certainwatermarking technologies. Watermarking techniques are also taught inthe following Philips patents: U.S. Pat. Nos. 6,252,972, 6,209,092,6,198,832, 6,157,330, 6,131,161, 6,031,815, 5,940,134, 5,933,798, and5,873,022.

[0005] Watermarking can be used to tag objects with a persistent digitalidentifier, and as such finds myriad uses. Some are in the realm ofdevice control—e.g., tagging video data with a do-not-copy flag that isrespected by compliant video recorders. (The music industry's SecureDigital Music Initiative (SDMI), and the motion picture industry's CopyProtection Technical Working Group (CPTWG), are working to establishstandards relating to watermark usage for device control.) Otherwatermark applications are in the field of copyright communication,e.g., indicating that an audio track is the property of a particularcopyright holder.

[0006] Other watermark applications encode data that serves to associatean object with a store of related data. For example, an image watermarkmay contain an index value that serves to identify a database recordspecifying (a) the owner's name; (b) contact information; (c) licenseterms and conditions, (d) copyright date, (e) whether adult content isdepicted, etc., etc. (The present assignee's MarcCentre service providessuch functionality.) Related are so-called “connected content”applications, in which a watermark in one content object (e.g., aprinted magazine article) serves to link to a related content object(e.g., a web page devoted to the same topic). The watermark canliterally encode an electronic address of the related content object,but more typically encodes an index value that identifies a databaserecord containing that address information. Application Ser. No.09/571,422 details a number of connected-content applications andtechniques.

DETAILED DESCRIPTION

[0007] EPGs organize and present (e.g., by time or topic) upcoming videoprogram content. As cable and satellite services increasingly havehundreds of channels, such guides become essential. EPGs are detailed ina variety of patents, including many assigned to Gemstar and StarSight,including U.S. Pat. Nos. 6,216,265, 6,118,492, 6,133,909, 6,144,401,6,167,188, 6,247,176, 6,151,059.

[0008] In typical EPG systems, a service provider (e.g., TV Guide)collects programming information from national, network and localprogram sources, and compiles it into a database. The database isindexed by geography and delivery source (e.g., Portland Oreg.; AT&TCable). Once a user's location and delivery source is known, thedatabase can be queried to identify programming that is upcoming, e.g.,for the next two hours. This data is typically presented in tabular(grid) form on the user's video screen. Advertising and otherinformation may be presented with the EPG data on the screen.

[0009] EPG data is presently conveyed to the consumer for displayon-screen through “out-of-band” techniques, such as the verticalblanking interval in analog video. Some systems have dedicated channelsthrough which such data is presented.

[0010] One drawback of such systems is their reliance on time-of-day asthe key by which program schedules are determined. If a sports game runslate, or if breaking news forces an interruption in normal programming,the EPG does not reflect the change.

[0011] In accordance with one aspect of the present invention, watermarkdata identifying a program is decoded from incoming video. Thisinformation is then checked against program identification data in theEPG and, if a discrepancy is noted, then a change in the programming isindicated on the displayed EPG. Consider Superbowl Sunday. The EPGdatabase may indicate that a SuperBowl broadcast on channel 2 isscheduled to end at 5:00 p.m. At 5:15, a watermark is decoded fromchannel 2 and conveys an identifier associated with the SuperBowl,indicating that the SuperBowl program has run past itsoriginally-allotted time. The EPG can update its entry for theSuperbowl, extending its ending time to 5:15 (the current time), 5:30(the next half-hourly interval), or otherwise mark it as changed. Datafor subsequent programming on that channel can likewise be treated aschanged (e.g., by pushing back all programming 15 minutes, or to thenext half-hourly interval, or otherwise), and displayed in the EPGaccordingly.

[0012] Continuing this scenario, assume the SuperBowl broadcast ends at5:20. This ending time can be detected by failure to detect theSuperBowl-identifying watermark from program material on channel 2 for apredetermined interval, such as 5 minutes. At 5:30, a new watermark IDis detected—this one corresponding to the program originally scheduledfor broadcast at 5:00. In this case, the database can shift by 30minutes the expected times of the 5:00 program, to 5:30. Viewers whorefer to the EPG at 5:35 will thus be presented with a timetable thataccurately reflects the currently available programs.

[0013] The expected timing of future programs can also be shifted in theEPG database and display, with the service provider deciding how far outin time to continue this shift. Certainly by midnight, for example, theprogramming would be expected to return to its originally scheduledtiming, with some compensating program change (e.g., an abbreviation ofthe evening news) making up for the SuperBowl overrun.

[0014] In some embodiments, the EPG interface presented to the userindicates that programming times for a particular channel are uncertain.In the example just given, for example, a graphical display of a programgrid for channel 2 may be highlighted in yellow from 6:00 p.m. untilmidnight, indicating that a change has made precise air-times uncertain.As each half-hourly slot arrives, however, the detection of a watermarkfrom the video then being sent permits at least the currently-availableprogramming to be accurately displayed. (Of course, a variety of othertechniques can be used to indicate schedule uncertainty, other thanyellow highlighting. Distinctive text presentation within the grid(e.g., italics), special effects (e.g., flashing text or backgroundcolor), textual alert messages, and a great variety of otherarrangements can be used to alert the viewer to the uncertainty.)

[0015] In addition to indicating uncertainty in the program schedule,the EPG presentation can also be altered to indicate that program timeshave been shifted from their expected values (i.e., certain times, butdifferent). Again, a variety of user interface techniques can be used tosignal this fact to viewers (including techniques detailed above).

[0016] Many VCRs, and essentially all Personal Video Recorders (PVRs),rely on EPG data to set start and stop times for recording functions.These devices can respond to watermarks, and/or EPG information that isupdated by reference to watermarks, to capture the desiredprogram—regardless of program delays.

[0017] Assume that the 5:00 program following the SuperBowl on channel 2is Seinfeld, and the user has earlier specified that Seinfeld (30minutes) should be recorded. At 5:00, the recording commences. Soon,however, the watermark data reveals that Seinfeld isn't yet playing. Outof abundance of caution, the device may continue to record. Or it maystop after, e.g., a five minute grace period, and try again, re-startingat the next half-hourly interval. Or, after stopping, it may immediatelyresuming recording if a Seinfeld watermark is detected between 5:05 and5:30.

[0018] If the device does not stop, but records continuously from 5:00onward, it may continue until 30 minutes after a Seinfeld-identifyingwatermark is first identified. Thus, if Seinfeld starts at 5:20, thedevice will continue to record until 5:50 (and optionally for a briefinterval after that time). If a Seinfeld watermark isn't detected withina predetermined window, e.g., 90 minutes, of the expected air-time, therecording device may conclude that the Seinfeld broadcast has beencanceled (avoiding recording of hours of unwanted programming).

[0019] In other embodiments, the stop time for a recording isn't set byreference to EPG data, or by reference to a known interval (e.g., 30minutes) after a start time. Instead, the device stops only when awatermark identifying a desired program is no longer detected. (Here, aselsewhere, “no longer detected” typically requires absence of detectionfor a period of several minutes, to account for commercials and otherinterruptions that may not convey the watermark of the desired program.)

[0020] If the recording device began recording at 5:00, it captured partof the Superbowl broadcast. Rather than save this unwanted programmaterial, it may be deleted. In PVRs, and other devices with digitalstorage, the storage space allocated to the unwanted programming cansimply be marked as un-used by the desired program, and returned to thepool of available storage. In one such embodiment, recorded programmingis discarded until a time a predetermined interval (e.g., 90 seconds)before first detection of the Seinfeld-identifying watermark.

[0021] In other arrangements, instead of deleting the non-Seinfeldprogram, it may be retained in storage. By reference to the updated EPGdata, or the watermark, the unwanted programming can be identified asthe end of the Superbowl. This information can be logged in the device'sindex of recorded material, together with an identification of itslength, and optionally the start and stop times of the originalbroadcast. (Other information, such as the location of the Superbowlvideo data in the device's file structure can also be maintained, butsuch information is typically transparent to the user.) When a table ofrecorded contents is presented to the user on-screen, the Superbowlexcerpt can be included among the listings—possibly set-off bydistinctive UI presentation to indicate that it was an unintendedcapture. When the user is first presented with this unintended programcapture, the system may ask the user whether it should be retained ordeleted. The system may have a default option, e.g., that unless theuser acts to preserve the accidentally-captured video, it is deleted.

[0022] In tape-based systems, if recording started at 5:00, and at 5:25the watermark corresponding to the recorded program still indicates anon-Seinfeld program, the tape may be automatically rewound to the pointwhere the 5:00 recording commenced. Then, at 5:30, recording cancommence anew, on the possibility that Seinfeld has been shifted to thenext half-hourly slot and will be properly captured by recording from5:30 to 6:00.

[0023] Apart from accurately presenting program information, andcorrectly recording desired programs, watermarking can be employed inconnection content security and authentication. Consider the delivery ofPay Per View (PPV) content. The PPV content may have usage rulesassociated with it. These rules may, e.g., disallow recording, or fastforwarding, or rewinding, or pausing, etc. These usage restrictions maybe conveyed by out-of-band or virtual channels, such as data transmittedprior to the video program, or during the vertical blanking interval, orin packet headers. In accordance with certain embodiments of the presentinvention, watermarks are used to represent this information.

[0024] For example, a video signal can include a watermark with an eightbit payload. The first bit, if set to “1,” can indicate no copying. Thesecond bit, if set to “1,” can indicate one copy permitted. The thirdbit, if set, can indicate pausing is permitted. The fourth bit cancorrespond to rewind permission, the fifth to fast forward permission,the sixth can indicate that any copy must be made on a certain type ofmedium (e.g., a local PVR), the seventh can indicate that any copy mustbe stored in encrypted form, etc. (If copy-once is permitted, the videocan be modified during the permitted copying operation to signal thatno-more-copies are authorized. This may be done, e.g., by applying afurther watermark to the signal.)

[0025] In still other embodiments, watermarks can be used in bandwidthoptimization strategies to provide augmented information, orprogramming, to a consumer. Consider a consumer who is interested inmotorcycling. This preference may have been specified explicitly by theconsumer, or may have been inferred through his behavior (e.g., hisobserved history of linking to on-line resources relating tomotorcycles). The topic of motorcycling may correspond to a particular16 bit identifier in a subject matter index (allowing 64 thousandsubjects). During hours when the PVR is not being actively used, it mayscan through all channels looking for material that is coded with themotorcycle subject code (perhaps among several others). If such materialis encountered, it is recorded, and a corresponding entry is made in thePVR's local table of contents. When the consumer next uses the device,he can see that a program of potential interest has been recorded.

[0026] The same technology can be used with advertising. Advertising canbe topically coded to identify the subject matter. If advertising isencountered having the motorcycle subject code, it can be captured andlocally stored for possible later display. In this case, thepresentation of the advertising can be requested by the user (as withany other recorded program), or the captured advertisement can beinserted in a commercial slot in other video programming (perhapssupplanting another commercial that is not as likely to capture theconsumer's interest).

[0027] Such encoding of video content with subject-indicating codes islimited by factors such as lengths of the codes, robustness tocompression/decompression and other distortion, video degradation, andother factors. In one system, a watermark with an 80 bit payload can beencoded in video, permitting recovery of 5 different content codesevery, e.g., 30 seconds. The same content codes can be repeated every 30seconds. Or, by repeating them less frequently, more codes can beconveyed (e.g., codes 1-5 in time 0-30 seconds; codes 6-10 in time 30-60seconds, codes 1-5 in time 60-90 seconds, etc.). Of course, thiswatermark may be overlaid or interleaved together with other watermarksconveying other information.

[0028] Instead of conveying absolute subject matter codes, each videoexcerpt can convey a unique ID that is used to access associated metadata in a data store. The data store may be local (e.g., downloaded to aset-top box periodically), or remote (e.g., at a cable head-end orelsewhere). Thus, a Seinfeld program may have a single code. But whenthat code is used to access a corresponding database record with metadata, the record may reveal 10 subject matter codes (e.g., comedy, NewYork City, Jerry Seinfeld, motorcycling, episode 29, episode title,etc.).

[0029] Different subject codes (or unique IDs) can be used for differentportions of a video program. So a ninety second clip that relates to amotorcycle may be coded to identify this subject matter, withoutso-coding the remainder of a program. The different portions need not bejust temporal portions. Through object segmentation technology, such asis used in MPEG-4, different on-screen objects can be encoded withdifferent watermark identifiers. Thus, a motorcycle in that ninetysecond clip may be encoded with a watermark indicating its motorcyclesubject matter, while a BMW automobile in the same scene may be encodedwith a different watermark.

[0030] In monitoring program material for desired codes, the consumerdevice can have a buffer in which the previous 15 seconds of video isalways available. Thus, if a desired watermark is detected, video from15 seconds prior to the detection can be written to long-term storage—toaccount for possible latency in watermark detection.

[0031] In a typical scenario, there may be 100 subject codes for which aconsumer's PVR is watching, corresponding to 100 subjects of potentialinterest to the consumer. As each watermark is detected, it is checkedagainst this list and, if a match is found, the video is captured(including the buffered 15 seconds prior to detection) for laterdisplay. If the program material is watermarked with unique IDs insteadof literal subject matter codes, the consumer device can query adatabase for the corresponding subject matter codes, and record thecontent if a match with one of the 100 profiled subject matter codes isfound. In some embodiments the database is maintained remotely,introducing a delay as the decoded codes are sent to the data, and theresults relayed back. Other approaches can mitigate this delay. Forexample, some (or all) of the database can be cached at the consumerpremises. Another approach is for the consumer device to periodicallysend its 100 searched-for subject matter codes to the database, whichthen returns a list of the Unique ID records for which the consumerdevice should be on the lookout (i.e., those having the subject mattercodes that are searched for).

[0032] The watermark detection functions referenced above can take placeat various different locations. In some embodiments, detection may takeplace at a device in the consumer home, such as in one or more of aset-top box, VCR, PVR, television monitor, etc. Information from suchdetection, in some embodiments, may be related back up the chain ofdistribution (e.g., a neighborhood distribution node, cable head-end,cable control center, national EPG database provider, etc.) Or thedetection can take place at any of the upstream locations. For example,a station through which all AT&T Cable signals destined for subscribersin Multnomah and Clackamas counties in Oregon can monitor all thosechannels.

[0033] In some applications, it is desirable to employ thewatermark-based systems detailed above in connection with known priorart techniques. PPV usage data, for example, can be conveyed both inheader data associated with encrypted content, as well as by watermarkdata. One data can be relied upon primarily (e.g., the header data) and,if absent or apparently corrupted, the watermark information can berelied upon instead.

[0034] As should be evident from the foregoing, certain embodiments ofthe invention contemplate that a unique watermark identifier isassociated with each video program. The identifier may be generic to aclass of programs (e.g., all Seinfeld shows are identified by the samewatermark), or each different program may have a different ID. In thelatter case, the watermark payload may have plural portions. One portionmay ID a family of programs (e.g., Seinfeld episodes), and anotherportion may convey an ID uniquely identifying a particular program inthat family (e.g., episode 88, “The Big Salad”).

[0035] The watermark identifier can be used to access a correspondingdatabase record where information about the identified program isstored. It may include the date the program was first broadcast (e.g.,“Sep. 29, 1994”), a synopsis of the program (e.g., “Jerry dates Newman'sex. George buys Elaine a big salad from the coffee shop, when hisgirlfriend takes credit for buying it. George lets Elaine know that itwas him that bought that salad. Show ends with Kramer driving Gendison'swhite Ford Bronco down the interstate (ala OJ )”), rights-holderinformation, digital rights management information, copy controlinformation, links to related programs, links to related merchandise,links to on-line resources including chat rooms and program archives,subject matter classification codes, etc. This database may be storedremotely from the user station, and remotely accessed by the user asnecessary. Or some or all of the database and contents can be kept (ormirrored) in a data store at the user's premises (e.g., in a set topbox). Such a database can be configured in accordance with a userprofile, e.g., specifying the class of programs to be detailed in thelocal database. Or the remote database can provide the local databasewith information corresponding to watermark IDs expected to beencountered in the next 7 days (e.g., during early morning hours whensystem traffic is otherwise low). A local user database can includeinformation provided by the user and not shared with a remote database,including private notes about likes/dislikes, etc. Or informationprovided by the user (including demographics and viewing habits) can bepassed to a remote database. Such personal information in the remotedatabase can be shared with other users, with a cable system operator,with the provider of the annotated database record, etc. The user mayreceive a fee in some circumstances for sharing such information.

[0036] To provide a comprehensive disclosure without unduly lengtheningthis specification, the patents and applications cited above areincorporated herein by reference.

[0037] Having described and illustrated the subject technologies withreference to illustrative embodiments, it should be recognized that theinvention is not so limited. For example, it will be recognized that theconcepts detailed above can be implemented with various forms ofwatermarking technologies, and can be advantageously combined instraight-forward fashion with other content-delivery systems. Moreover,it will be recognized that wherever in the prior art that verticalblanking interval-based data communications techniques were used, thesame applications may be served instead by conveying such data throughin-band video watermarking. Further, it should be recognized that theparticular combinations of elements and features in the above-detailedembodiments are exemplary only; the interchanging and substitution ofthese teachings with other teachings in this and theincorporated-by-reference materials are also contemplated.

We claim:
 1. A method of controlling operation of a video recordingdevice to record a desired video program, comprising: monitoring in-bandvideo watermark information conveyed with incoming video to identify aprogram; and when the desired video program is identified, recordingsaid incoming video.
 2. The method of claim 1 that includes terminatingthe recording of said incoming video after watermark informationidentifying the program is not detected for an interval.
 3. The methodof claim 1 that includes buffering the incoming video prior to saidrecording, and including at least a portion of said buffered video withthe recorded video so that an initial portion of the video program isnot lost.
 4. A consumer video appliance for recording a desired videoprogram, comprising: a watermark detector for monitoring in-band videowatermark information conveyed with incoming video to identify aprogram; and a storage writer operative to record the desired videoprogram when the watermark detector identifies the desired program.
 5. Amethod of controlling operation of a video recording device to record adesired video program, comprising: at an expected commencement time ofsaid video program, starting recording; monitoring an in-band videowatermark in the video program being recorded; and if a video watermarkcorresponding to the desired video program is not sensed within aninitial interval, discontinuing said recording.
 6. The method of claim 5that includes: after said discontinuation of said recording, continuingto monitor the in-band video watermark, and resuming said recording if avideo watermark corresponding to the desired program is detected.
 7. Themethod of claim 6 that includes continuing said resumed recording for aninterval approximately equal to an expected duration of said videoprogram.
 8. The method of claim 5 that includes, after saiddiscontinuation of said recording, resuming said recording at a nexthalf-hourly interval.
 9. A method of ending the recording of a desiredvideo program, comprising: monitoring an in-band video watermark in thevideo program being recorded; and at an expected termination time ofsaid video program, stopping said recording, unless an in-band videowatermark corresponding to the desired video program is sensed,indicating that the program is continuing beyond the expectedtermination time, in which case the recording is permitted to continue.10. The method of claim 9 that includes continuing said recording untilan in-band video watermark corresponding to the desired video program isno longer detected.
 11. A method of ending the recording of a desiredvideo program, comprising: monitoring an in-band video watermark in thevideo program being recorded; and ending said recording when an in-bandvideo watermark corresponding to the desired video program is no longerdetected.
 12. A method of operating a video recording device,comprising: storing data indicating viewing interests of a user;monitoring in-band video watermark data conveyed by incoming video; andrecording incoming video determined to be of potential interest to theuser by reference to the in-band video watermark data therein, and thestored data.
 13. The method of claim 12 that includes adding informationabout a video thus-recorded to an electronic table of contents ofprogramming recorded by the video recording device.
 14. The method ofclaim 12 that includes scanning through plural sources of incoming videoand monitoring same.
 15. The method of claim 12 in which the videowatermark data includes at least one subject matter code relating tosaid incoming video.
 16. The method of claim 12 in which the videowatermark data includes index data that is used to access acorresponding store of information that includes subject matterinformation about the incoming video.
 17. The method of claim 16 inwhich the store is maintained at a user premises.
 18. The method ofclaim 16 in which the store is maintained remote from a user premises.19. The method of claim 12 in which said data is stored at a locationremote from the user premises.
 20. The method of claim 12 in which saidmonitoring occurs in a device at the user premises.
 21. The method ofclaim 12 in which said monitoring occurs remote from the user premises.